Skip to content

Fix discovery bootstrap cache access and failure diagnostics - #409

Draft
almaslennikov wants to merge 1 commit into
mainfrom
codex/issue-408-bootstrap-diagnostics
Draft

almaslennikov wants to merge 1 commit into
mainfrom
codex/issue-408-bootstrap-diagnostics

Conversation

@almaslennikov

@almaslennikov almaslennikov commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

The bootstrap daemon can become Ready before NIC initialization finishes, then crash while Launch Kit shows only the device-count wait. Its cluster-wide ConfigMap cache also lacks list/watch permissions in Launch Kit's bootstrap role.

This change grants ConfigMap get/list/watch without write verbs, reports unverified eligibility-probe exclusions by node/pod, and refreshes bootstrap pod failure evidence for missing publishers during the NicDevice wait. Diagnostics are deduplicated, deadline-bound and nonfatal so a restarting publisher can recover. Timeout errors name the last missing publishers, and progress counts only the expected node set. Existing-daemon/read-only waiters acquire no pod-status operations.

Related to #408. This is a scoped correction, not a complete fix for readable ConnectX inventory being lost when BlueField startup discovery aborts. The existing trust probe and SKIP_DEVICE_ON_DISCOVERY_ERROR policy remain unchanged.

Evidence and compatibility

  • Verified the reported Launch Kit base 6c54e0f870f0d4ccc91fe4a5ec84440d86e587dd, merged fix: exclude restricted BlueFields from discovery #246/fix(discovery): report host-restricted BlueField exclusions #341, and all issue comments.
  • Release 26.7 resolves to the reported network-operator-v26.7.1 daemon tag. Its tagged source registers a cluster-wide ConfigMap cache; bootstrap's direct-Get permission cannot satisfy its informer.
  • The same tag's skip option can omit ConnectX firmware/PSID failures as well as BlueField mode failures. Its skipped-device list is in-process only; the controller can preserve existing CRs for omitted devices. Enabling it alone could turn missing/stale topology into a successful node-level wait.
  • A failed trust query remains unknown, not proof of permanent restriction. NICO #601 is awaiting a reliable restriction signature. The reduction from four GPU nodes to one expected publisher is consistent with failed exec probes, but requires reporter evidence to establish the exact cause.

Updated the existing discovery and troubleshooting guides, their skills/references, and the discovery-inventory contract. Regression cases cover exact read-only RBAC, live crash/recovery, repeated failure suppression, replacement pod UID, namespace/owner/pending-node scope, successful init completion, cancellation/deadlines, primary-error preservation, missing-node errors and unchanged read-only polling.

Validation

Local macOS/arm64, Go 1.26.2:

  • make build: passed.
  • make test: passed. The initial sandbox denied local httptest listeners; rerunning with loopback access passed.
  • make lint, pinned golangci-lint v2.14.0: zero issues.
  • go test -race -count=1 ./pkg/networkoperatorplugin/discovery ./pkg/nicconfigdaemon: passed on the final code.
  • python3 scripts/check-docs.py --binary build/l8k: passed, including 247 CLI/help examples, navigation and offline rendering.
  • mkdocs build --strict: passed using the pinned documentation requirements in an isolated virtual environment.
  • git diff --check: passed.
  • go test -race -timeout=45m -count=1 ./...: passed. The final affected-package race run above also covers the completed-init and cancellation regression cases added during final review.

Two independent final code/design reviews approved this scoped correction. Hosted CI build (including make build-all), lint, documentation and CodeQL checks have passed. The hosted Linux/Go 1.27 race-and-coverage test job is still running; this is separate from the completed local race checks.

Draft review questions and remaining evidence

  1. Confirm cluster-wide ConfigMap read access is acceptable for the bootstrap service account. Kubernetes RBAC cannot restrict these reads to the daemon cache's label filter. Namespace-scoped permissions cannot satisfy this binary's cluster-wide list.
  2. Confirm warning-and-recovery behavior during the existing wait is preferred to aborting on crash loops. This change deliberately preserves the wait policy and prints pod status plus commands to retrieve current/previous logs, rather than automatically fetching raw logs.
  3. Decide the follow-up partial-inventory contract before enabling skip-on-error: a typed omission/completeness signal, or a deliberate opt-in with per-device reporting and stale-CR handling. A blanket failed-trust-query exclusion is unsafe without a reliable restriction signature.

The narrower RBAC-only alternative would leave the reported opaque wait and node-count mismatch undiagnosed. The observer keeps those diagnostics local to bootstrap without changing shared read-only discovery behavior.

No live cluster, GB300/arm64 or BlueField validation was performed. Image digest-to-source provenance, actual cache authorization/synchronization, affected hardware behavior and the exact dropped-node cause remain unverified. Hosted CI and cross-platform qualification are separate from local checks. Keep this PR draft until reviewers agree on the read-permission scope and diagnostic approach and the relevant runtime evidence is assessed.

Grant the bootstrap daemon the ConfigMap read verbs required by its cache. Report unverified NIC probes and pending-publisher failure evidence without changing trust classification or device omission policy.

Related to #408.

Signed-off-by: Alexander Maslennikov <amaslennikov@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant